Malicious software applications (i.e., malware) can perform a wide range of harmful actions, such as theft of passwords and other confidential user data, accessing financial data to commit acts such as identity theft and other fraud, recruitment of the computer into a network of bots to perform denial-of-service (DoS) attacks or to send spam, blockage of system functionality for extortion purposes, among many other malevolent actions.
In order to infect a computer system and avoid detection by antivirus programs, malware is typically embedded into seemingly benign programs that appear to perform some useful function. During execution of such programs, the malware code is designed to be executed unbeknownst to the user. Massive effort has been expended to-date to combat malware. However, malware developers continuously find new ways to avoid computer security systems. Thus, a data security arms race persists in which malware and anti-malware developers are engaged in a repeating cycle of creating new “weapons” and defenses to counter those weapons.
Computer software is a complex set of instructions involving transfers to different execution environments, system calls, cycles, etc. It should be noted that the complexity of executable files is constantly increasing, which is attributable to the growing popularity of high-level programming languages and to the increasing complexity of computer equipment and operating systems. This is true for both trusted and malicious applications.
One of the techniques employed to evaluate a potentially malicious program is emulation of the execution of the program by the antivirus application, which permits the antivirus application to observe and analyze the program's behavior. Emulation involves software-based modeling, or imitation, of a computer's components—processor, memory and other system devices—by creating virtual representations of processor registries, memory, and the processor instructions set, and parts of the operating system. In this manner, the program instructions are executed not on the actual processor and related hardware, but on the virtual copy of these components in a protected, isolated, environment.
In order to counter the program code emulation, creators of malicious programs use various tricks, which can be based on the limitations related to the emulation process and to the realization of an emulator in antivirus solutions. One of these approaches involves adding to the program code a large number of instructions, which do not carry a malicious component but require time for emulation. Considering the fact that the time allocated for the emulation of a program code is limited to prevent user discontent (usually, this time can be a few seconds), the emulation process can stop before the execution of a really malicious code. One of the methods to fight such anti-emulation approach is described in the U.S. Pat. No. 7,603,713, the disclosure of which is incorporated by reference herein, the operation mode of which includes execution of a number of instructions on an actual processor, substantially accelerating the process of emulation of unknown applications in order to reach the malicious code itself. Another approach, disclosed in U.S. Pub. No. 2011/0225655, the disclosure of which is incorporated by reference herein, considers the option of defining an application as suspicious if it resists emulation. Such approach is based on the fact that a safe (legitimate) application usually displays a warning about the impossibility of execution in the emulator, while a malicious one comfortably completes the execution or suddenly changes the execution vector (logic).
Another anti-emulation method is based on the determination of the fact that the application is currently launched specifically in the emulator, and not in an actual operating system. It should be noted that an emulator creates a virtual representation of a processor, computer components and operating system (OS) only to a limited extent, with reduced capabilities. Given the complexity of operating systems, calls made by applications to the operating system's services via the application program interface (API), such as calls utilizing subsystem dynamic link libraries (DLLs), which in turn provide protected access to the functionality of the operating system's kernel via the ntdll.dll file, for example, are emulated by returning a fabricated (i.e., “emulated”) result of that API call. In other words, during emulation, such system calls are not actually processed. Therefore, avoiding “full system” emulation provides a substantial savings in complexity and time of execution for the emulation, which is important to preserve a reasonably fast and responsive user experience. Indeed, a complete reproduction of all the capabilities of the processor or system API functions is impractical for several reasons—high labor costs for such design, the presence of undocumented functions, and dramatic loss of productivity during the operation of such emulator.
Exploiting this knowledge, creators of malicious programs can use the following techniques, which allow the program to discover the fact of execution in an emulated environment:                Calling an undocumented or rarely used API function.        Execution of a number of instructions by a central processor with subsequent check, for example, of a number of set flags. In case of insufficiently accurate emulation of processor commands, a number of flags can have values which differ from the ones that would be set in case of execution on an actual OS.        Verification of the correctness of API function execution. Such verification can be very complicated and can include analysis of returned error codes in case of incorrect call or verification of the set processor flags.        A memory scan for presence of specific bytes associated with API call processing is also a popular anti-emulation method.        
The latter approach is widely used because it allows to constantly bypass the existing emulators, as it is impossible to implement a correct emulation of each API function for the above-described reasons.
In view of the above, a practical solution is needed for defeating anti-emulation techniques.